This study aims to investigate the use of refractory brick (RBA) by-products as a substitute for coarse aggregate in sustainable concrete production. Concrete mixes with water-to-binder (w/c) ratios of 0.52 and 0.49 and containing 0%, 15%, 30%, and 50% RBA as a partial replacement for ordinary crushed stone (OCS) were produced. The following properties were examined in this study: workability, COMPRESSIVE strength, stress-time relationship, TOUGHNESS index, performance criteria, and cost analysis. The test results showed that an increase in the RBA percentage in the concrete mixtures positively contributed to concrete workability. Moreover, the COMPRESSIVE strength and all phases in the COMPRESSIVE stress and time relationship decreased as the percentage of RBA in the concrete mixture increased. However, at 15%RBA, the TOUGHNESS index value was comparable to that of the reference concrete, whereas based on the performance criteria, the replacement of OCS with 15%RBA for both water-cement ratios met the minimum requirements. Meanwhile, cost comparison analysis discloses that material and production costs can be reduced by approximately 51.84% and 1.5–6.5%, respectively. Based on the analysis of all the test results, 15%RBA exhibited insignificant differences in value compared with the reference concrete. Thus, the use of 15%RBA as an OCS replacement is an acceptable and viable option for producing sustainable concrete.